Automatic keyphrase labeling using search queries

ABSTRACT

Automatic keyphrase labeling and machine learning training may include a processor extracting a plurality of keywords from at least one search query that resulted in a selection of a document appearing in a search result. For each of the plurality of keywords, the processor may determine a probability that the keyword describes the document. The processor may generate one or more keyphrases by performing processing including selecting each of the plurality of keywords having a probability greater than a predetermined threshold value for insertion into at least one of the one or more keyphrases and assembling the one or more keyphrases from the selected plurality of keywords. The processor may label the document with the keyphrase.

BACKGROUND OF THE DISCLOSURE

A keyphrase is a phrase that explains a longer document. For example, akeyphrase can be a word or short phrase (e.g., 1-10 words, one sentenceor clause, etc.) that describes a main idea of an article, book,website, or other document. One or more keyphrases may describe theessence of a document and/or provide a summary of the document.Keyphrase extraction aims at automatically selecting a small set ofphrases in a document that describe the document's main ideas.Extracting keyphrases that describe documents is a fundamental task innatural language processing (NLP) that can provide keyphrases for manyapplications (e.g., search engine optimization, document categorization,site navigation, etc.).

Broadly speaking, automatic keyphrase extraction methods described inthe literature can be categorized into supervised and unsupervisedlearning approaches. Most keyphrase extraction methods use supervisedmachine learning and/or manual work by human experts. Even whensupervised machine learning algorithms for keyphrase extraction areused, these algorithms require labeled data. Labeling is done by humanannotators who label keyphrases in a corpus. The downside is the needfor tedious and expensive hand-labeling. Furthermore, this costlyoperation must be repeated for each domain. Obtaining domain-specificlabels requires extensive and inefficient manual work, and furtherrequires storage, maintenance, and updating of a large corpus of labeleddata. Moreover, manual work, or even training a supervised model onmanually labeled data, introduces the possibility of error. Meanwhile,unsupervised approaches for keyphrase extraction can be unreliable inproducing usable results.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an automatic keyphrase labeling system according to anembodiment of the present disclosure.

FIG. 2 shows a keyphrase labeling and machine learning computingenvironment according to an embodiment of the present disclosure.

FIG. 3 shows keyword extraction processing according to an embodiment ofthe present disclosure.

FIG. 4 shows probability determination processing according to anembodiment of the present disclosure.

FIG. 5 shows keyphrase assembly processing according to an embodiment ofthe present disclosure.

FIG. 6 shows document labeling processing according to an embodiment ofthe present disclosure.

FIG. 7 shows a computing device according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION OF SEVERAL EMBODIMENTS

Embodiments described herein may use one or more automated techniques toautomatically label documents for use by keyphrase extraction modeltraining algorithms. These techniques can allow machine learning (ML)models to be trained and perform well on data that has not been labeledby human experts or that may be continuously evolving and changing. Thisprocess may leverage logged data of a deployed search engine. Forexample, users enter search queries into a search engine, and the searchengine returns results. Users examine the results and select one or moreresults that they believe to be relevant to their query. As searches areperformed and results are selected by a large body of users over aperiod of time, a large corpus of logged search engine data may beassembled. Based on the intuition that a user tells a search engine theinformation he needs and selects a search result that meets the need, itmay be assumed that a selection by a user indicates the selected resultis relevant to the query. Based on this assumption, embodiments mayextract keyphrases from searches and correlated them with informationfrom the results of those searches.

In comparison with expert-prepared labels, labels generated byembodiments may be relatively noisy and may be more or less accurate as.However, even in these cases, the massive volume and continuousrefinement of these generated labels may allow ML systems to performlearning based on these labels (e.g., supervised learning). Moreover,labeling by an ML system trained on search engine data can be betterthan that performed by experts because it can automatically leverage thewisdom of crowds. The history of user interactions in the search enginedata illustrates an actual relevance of data to keyphrases. In thissense, embodiments described herein may not only automate a process, butmay also improve the accuracy and quality of process outputs relative toboth human-centric and automated alternatives.

FIG. 1 shows an automatic keyphrase labeling system 100 according to anembodiment of the present disclosure. A search query 101 may include oneor more keywords that, through processing by system 100, may form akeyphrase. A user may have entered query 101 into a search engine andreceived, as one of the search results, document 102 (or a link or otherreference to document 102). System 100 may use query 101 and document102 to produce labeled document 103. Labeled document 103 may includedocument 102 with a keyphrase label associated therewith, so thatlabeled document 103 may be used as training data for one or moresupervised ML algorithms. System 100 may determine a keyphrase fromquery 101 and use the keyphrase to label document 102, thereby producinglabeled document 103, as follows.

System 100 may receive query 101, where query 101 is a search query thatresulted in a user selecting document 102 (e.g. by clicking on a link todocument 102 in the search results, downloading document 102, etc.).System 100 may only process queries 101 that result in selections of atleast one document 102, because queries 101 where documents 102 are notselected have not generated any relevant information describing thecontent of any documents 102. System 100 may perform keyword extractionprocessing 110 on query 101, which is described in detail below withrespect to FIG. 3. In summary, for a query 101 containing multiple wordsand/or characters, system 100 may remove some words and/or charactersfrom query 101 depending on its content (e.g., removing stop words,punctuation, etc.), such that the remaining portions of query 101 (e.g.,one or more keywords) may be processed separately from the removedportions, as described below.

System 100 may process the one or more keywords from keyword extractionprocessing 110, along with document 102 itself, to evaluate whether thekeywords describe document 102. For example, system 100 may performprobability determination processing 120, which is described in detailbelow with respect to FIG. 4. Probability determination processing 120may include determining an affinity of query 101 to document 102 interms of selection frequency, determining probabilities that eachkeyword describes document 102, and/or determining probabilities thatportions extracted from query 101 during keyword extraction processing110 describe document 102.

The results of probability determination processing 120 may informsubsequent processing to assemble a keyphrase relevant to document 102.For example, system 100 may use the results of probability determinationprocessing 120 as inputs to keyphrase assembly processing 130, which isdescribed in detail below with respect to FIG. 5. Keyphrase assemblyprocessing 130 may include selecting keywords that describe document 102with high probability (e.g., probability greater than a predeterminedthreshold), assembling them into a keyphrase, and determining aprobability that the keyphrase itself describes document 102.

System 100 may perform document labeling processing 140, described indetail below with respect to FIG. 6, to label document 102 with akeyphrase developed through keyphrase assembly processing 130. Forexample, assuming a keyphrase was found that describes document 102 witha high degree of probability (e.g., probability greater than apredetermined threshold), document 102 may be labeled with thiskeyphrase, allowing labeled document 103 to be used for ML training byML system 200. This labeling is automatic and leverages the wisdom ofthe crowd in a way that basic “expert” labeling cannot, therebyproducing training data not only quickly and efficiently, but also withan improved level of insight and accuracy that cannot be replicated byeven the most skilled human expert.

FIG. 2 shows a keyphrase labeling and machine learning computingenvironment 10 that may leverage system 100 according to an embodimentof the present disclosure. The computing environment 10 may include oneor more devices in communication with one another through a network 12.Network 12 may include any public and/or private network, such as theInternet. The devices may include automatic keyphrase labeling system100, a query system 14, and/or a machine learning system 200. One ormore of these systems may include and/or be in communication with datastorage such as query database 16, document corpus 18, and/or trainedmodels 20. These systems (including storage elements) are illustrated asseparate components connected by network 12 in the example of FIG. 6,but in some embodiments, each system (or some of the systems) maycollectively be elements of a single device and/or may be coupled to oneanother by direct connections rather than by network 12. Likewise, insome embodiments, each system (or some of the systems) may bedistributed among multiple devices. An example computing device that mayprovide some or all elements of FIG. 2 is described below with respectto FIG. 7.

In the example of FIG. 2, text that is analyzed by automatic keyphraselabeling system 100 may be supplied by query system 14 and querydatabase 16. For example, query system 14 may be a search engine, andqueries (e.g., query 101) and results of queries (e.g., includingdocuments 102) performed by query system 14 may be stored in querydatabase 16. In a specific example for ease of explanation, query system14 may be a device that provides search capabilities for an onlineapplication such as a tax preparation application or accountingapplication.

Query system 14 may provide data (e.g., query 101 and document 102) fromquery database 16 to automatic keyphrase labeling system 100 (e.g.,through network 12), or in other embodiments automatic keyphraselabeling system 100 may access query database 16 directly. Automatickeyphrase labeling system 100 may process the text as described below(e.g., with respect to FIGS. 3-6). The output of such processing mayinclude a plurality of labeled documents 102.

Labeled documents 102 may be used for any purpose. The environment 10 ofFIG. 6 provides one example, which is a machine learning system 200. Inthe environment 10 of FIG. 6, automatic keyphrase labeling system 100 isconfigured to store labeled documents 103 in document corpus 18. Machinelearning system 200 may train a supervised learning algorithm ondocument corpus 18, yielding a trained model 20. Machine learning system200 may use any supervised learning algorithm known to those of ordinaryskill in the art or developed in the future. For example, as discussedbelow, this allows automatic keyphrase labeling system 100 to functionas an automatic training data creation system producing data withoutexpert input that has advantageous training features relative to workproduced by experts. Thus, a machine learning system 200 using thedocument corpus 18 for training may build more effective trained models20 than other ML systems that use expert-produced corpus data alone. Thetrained model 20 may subsequently be used by machine learning system 200to automatically identify keyphrases for unlabeled documents.

FIG. 3 shows keyword extraction processing 110 according to anembodiment of the present disclosure. A query 101 may include multiplewords and/or punctuation elements. In cases where there are multiplewords and/or punctuation elements (e.g., as opposed to a single-wordquery 101), keyword extraction processing 110 may separate query 101into portions that are likely to be specifically descriptive of document102 on their own (e.g., keywords) and other portions that convey lessmeaning on their own (e.g., very common or “stop” words or phrases,punctuation).

At 202, system 100 may remove at least one uninformative portion fromquery 101, so that the remaining portion(s) of query 101 are allkeywords. Generally, search queries 101 may concisely describe a user'sinformation needs. Queries 101 are relatively short, and a large portionof their tokens (e.g., words and punctuation) is informative. Therefore,it may be computationally and logically more efficient for system 100 toextract keywords from queries 101 than from entire documents 102. Forrelatively simple queries 101, such as those that contain a singlesentence, the extraction process may simply omit any (relatively few)uninformative tokens using a small handful of intuitive rules. Forexample, system 100 may apply rules such as omitting the following:tokens with dependency edges that are either auxiliary (e.g., has orshould) or negation, punctuation marks, and/or stop words (that are notphrasal verb particle like blow *up*). For more verbose queries 101,system 100 may apply one or more known techniques for keywordextractions (e.g., techniques discussed in “Discovering Key Concepts inVerbose Queries,” Michael Bendersky and W. Bruce Croft, Proceedings ofthe 31st Annual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, A C M, 2008, the entirety of whichis incorporated by reference herein).

At 204, system 100 may aggregate query 101 with like queries 101 ifapplicable. Since queries 101 are used as side information to the endgoal of document keyphrase extraction, all queries 101 with the exactset of keywords or keyphrases may be treated as a single equivalenceclass. That is, queries 101 that yield the same keywords or keyphrasesmay be merged to be a single meta-query. Herein, after processing at204, the term “query 101” may refer to a standalone query 101 receivedby system 100 or a combined meta-query, as both types may be processedsimilarly. For example, system 100 may have performed keyword extractionprocessing 110 on other queries 101 in the past. In some cases, query101 being analyzed may be the same as a previously-analyzed query 101after uninformative portions are removed. For example, a previous searchquery 101 may have been “a black dog” and the current search query 101may be “the black dog,” and both queries 101 may have resulted inselection of a document 102. Both queries 101 may be collapsed to“black” and “dog” as the keywords (which may be ordered keywords forminga keyphrase, as discussed below). Accordingly, both queries 101 may beaggregated into a single query 101 for further processing according tothe processes of FIGS. 3-6 and/or other processing described herein.System 100 may store a count of queries 101 that have been aggregatedand, whether aggregated or not, an indication of which document(s) 102were selected in response to the queries 101 (e.g., a meta query mayinclude 30 separate queries 101 with the same keywords, and the separatequeries 101 may respectively have resulted in the selection of same ordifferent documents 102), which may be used for some other processing insome cases as described below.

At 206, system 100 may store removed portions removed from query 101 forlater use (e.g., within probability determination processing 120).

System 100 may process the keywords determined as described above (and,in some cases, extracted portions stored at 206) using probabilitydetermination processing 120. FIG. 4 shows probability determinationprocessing 120 according to an embodiment of the present disclosure.System 100 may perform probability determination processing 120 todetermine probabilities that individually extracted keywords, and/orkeyphrases comprising a plurality of the extracted keywords, describedocument 102. As noted above, system 100 may consider only queries 101that result in the selection of at least one document 102, but theselection might be a mistake, or the user might have decided afterreviewing a selected document 102 that the document is not relevant tothe query. These cases represent false positives, and the query 101 isnot descriptive of the selected document 102 in these cases.Accordingly, system 100 may perform probability determination processing120 to guard against false positives.

Probability determination processing 120 may have multiple components,including a component whereby system 100 determines an affinity of thequery 101 to the document 102. The affinity may form a component of eachprobability of each respective keyword in the query 101. Given loggedinformation of a search engine, it may be possible to infer the affinitybetween a query-document pair (q, d). For example, this may be done byconsidering the amount of clicks on d as a result of q and an amount oftimes where d was shown as part of the result set of q but was notclicked, as performed at 302-306 of probability determination processing120. Alternatively, or additionally, system 100 may consider the averagedwell time of users on document 102 after clicking through from a searchresult driven by query 101 as an indicator of affinity.

At 302, system 100 may determine an expected random distribution ofselections of the document 102 in the search result generated byinputting query 101 among selections of any of a plurality of documentsappearing in the search result. To determine the expected randomdistribution of selections, system 100 may construct a model under whichall clicks are randomly proportional to popularity using statisticsprovided by the search engine to which query 101 was input. Thesestatistics can include data such as a number of times a query was runand numbers of times each result provided in response to the query wasselected. For example, assume there are two documents returned inresponse to query 101, doc1 and doc2. In this example, doc1 was clicked1 million times, and 33% of users clicked on doc1 as their firstselection. In the same example, doc2 was clicked 2 million times, and66% of users clicked on doc2 as their second selection. Accordingly, ifquery 101 (a meta query) is asked 30 times, the model postulates thatselection of doc1 should happen 10 times, and selection of doc2 shouldhappen 20 times.

At 304, system 100 may determine an actual distribution of selections ofthe document 102 from the statistics provided by the search engine. Asnoted above, system 100 may maintain a record of document(s) 102selected in response to a query 101. Continuing the example above, query101 was asked 30 times, resulting in selection of doc1 29 times and doc2one time.

At 306, system 100 may set the affinity of query 101 to document 102based on a proportion of the actual distribution to the expected randomdistribution. For example, system 100 may determine a normalapproximation to a binomial distribution to compute a probability of avalid connection between query 101 and document 102. Continuing theexample above, the observed vs. expected binomial distribution for doc1is 29 clicks vs. 10 expected clicks. The probability of thisdistribution is low (i.e., 29 clicks is much higher than expected),suggesting a valid connection between query 101 and doc1 (i.e., document102). Taking an inverse of this binomial distribution probability mayyield the probability that the connection is relevant (i.e., that theaffinity exists). System 100 may determine whether this probability ofaffinity is greater than some predetermined threshold for relevance and,if so, determine that there is an affinity between query 101 anddocument 102.

Probability determination processing 120 may also have a componentwhereby the respective relevancies of individual keywords to document102 are determined. When keyphrase assembly processing 130 is performed(for example, as described below with respect to FIG. 5), system 100 mayuse both the probability of affinity calculated for query 101 anddocument 102, and the individual probabilities of respective keywords,to determine whether a keyphrase accurately describes document 102.Accordingly, for each of the plurality of keywords, system 100 maydetermine a keyword component of the probability based on a comparisonbetween the keyword and content of the document 102, for example asperformed at 308-312 of probability determination processing 120.

At 308, system 100 may, starting with one of the keywords from query101, find a probability that the keyword describes the document 102,which may be a keyword component of an overall probability. For example,system 100 may search document 102 for the keyword. If the exact keywordis found in document 102, system 100 may assign a 100% probability ofrelevance to that keyword. For at least some words, system 100 maymaintain a set of synonyms and/or similar words. System 100 may searchfor these synonyms and/or similar words as well. For example, “boy” maybe a keyword, and system 100 may find “child” in document 102 and,because “child” is listed as a similar word to “boy,” may assign a 100%probability of relevance to that keyword. Alternatively, or in the casewhere there is no match, system 100 may use a word embedding approach toanalyze probability. For example, system 100 may use word2vec or anothervectorization algorithm on the keyword and the document 102 and performunsupervised training on the document 102 as a corpus. System 100 maydetermine whether any of the vectors in document 102 have a similarityto the vector of the keyword (e.g., above some threshold such as 0.5).For example, this may locate similar vectors such as a vector for“reconciliation” in the keyword and a vector for “reconcile” in document102.

At 310, system 100 may determine whether any keywords remain whoseprobabilities have not been calculated. If so, at 312, system 100 mayselect another keyword from query 101 and process the keyword asdescribed above at 308.

Once individual keyword probabilities for each keyword have beendetermined, system 100 may form a combination probability for eachkeyword that uses both the affinity between the search query and thedocument and the keyword component of the probability. For each keyword,at 314, system 100 may determine this probability by multiplying theaffinity probability and the keyword component together, for example.

At 316, system 100 may determine probabilities for the removeduninformative portions of the query 101 obtained at 202 in keywordextraction processing 110. To do so, system 100 may apply at least thefollowing two rules. A first rule may apply in the case of two keywordsdetermined to relate to document 102 above that have at least oneuninformative portions interposed between. In this case, the probabilityof each interposed uninformative portion is determined as a function ofthe two keywords it is between. For example, for two keywords withscores s1 and s2, the score of each intermediate uninformative token maybe given by min(s1, s2). A second rule may remove singular keywords. Forexample, if a token is marked as a keyword, but its surrounding tokensare not (i.e., are uninformative removed portions), then it may beremoved as a keyword regardless of its score, along with the surroundingtokens.

FIG. 5 shows keyphrase assembly processing 130 according to anembodiment of the present disclosure. After keywords have been assignedprobabilities by probability determination processing 120, system 100may perform keyphrase assembly processing 130 to construct keyphrasesthat describe document 102 with high probability.

At 402, system 100 may select high probability keywords as determined byprobability determination processing 120 for inclusion in the keyphrase.For example, all keywords and extracted portions with probability scoresthat exceed some threshold may be finally marked as keywords for thepurpose of assembly into a keyphrase.

At 404, system 100 may assemble keywords as marked at 402 into akeyphrase. Consecutive keywords may be assembled into a single keyphrasein the order in which they originally appeared in a query 101. Forexample, query 101 may have been “What is the airspeed velocity of anunladen swallow?” The first three words and the final punctuation markmay have been eliminated as being uninformative portions (stop words andpunctuation) not surrounded by keywords. Thus, the keyphrase thatresults is “airspeed velocity of an unladen swallow” with fourdescriptive words (airspeed, velocity, unladen, swallow) and two stopwords that are surrounded by descriptive words (of, an), in the order inwhich they appeared in query 101.

At 406, system 100 may determine an overall probability that thekeyphrase describes the document, for example by multiplying theprobabilities for each word in the keyphrase together.

At 408, system 100 may apply a decay function to the probabilitydetermined at 406. For example, the decay function may allow therelevance of the keyphrase to adjust as a document 102 changes inpopularity. This may be based on the assumption that the a prioriprobability of a token to be a keyword is independent from the number ofclicks (popularity) of the document. However, as observed from the data,there can be a positive correlation between document popularity andproportion of keywords. This correlation can be explained by the factthat unpopular documents lack queries that could provide evidence fortokens being keywords. Therefore, system 100 may introduce the notion ofa yet unobserved query which would lead to a click on a specificdocument and mark a specific token as a keyword. Intuitively, it may besaid that if a document is already popular, then the existenceprobability of an unobserved query is low, and vice versa. System 100may use a sigmoid function that decays to zero so that as popularityincreases, system 100 can decrease the probability of having a hiddenquery on the document. This may be done because there may be a positivecorrelation between document popularity and proportion of keywords. Thiscorrelation can be explained by the fact that unpopular documents lackqueries that could provide evidence for tokens being keywords.

Mathematically, the above-described probability determination processing120 and keyphrase assembly processing 130 may be restated as follows.

The objective is to utilize the information in queries in order to labelkeyphrases in the associated documents. To achieve this, system 100 mayassess the affinity between each query-document pair. Denote Q_(d) theset of queries that led to clicks on a document d. A query q∈Q_(d) maybe associated with an erroneous or random click, and in that case maynot in fact contain relevant information. Denote the probability that adocument d is indeed relevant to a query q with P_(r)(d|q).

Given that d is relevant to q, it can be asserted that if q appears in das a sub-string, then it is a keyphrase in d. These may not be all thekeyphrases, for instance since the same idea may appear in differentwording. System 100 may use click data to widen the search, based on abag-of-words approach. For each nonstop-word token t E d, system 100 maycompute the likelihood of it being part of a keyphrase. Denote the eventof having t as a keyword in d with kw_(d)(t)¹. One task of system 100may be to assess this probability.

The token-level granularity used by system 100 may allow system 100 toovercome the problem of exact string matching by computing theprobability that a token t appears in a clicked document d is relevantto a query q. Denote the probability of this event P_(r)(t|q). Eachquery q∈Q_(d) is an observation that contributes information regardingthe event kw_(d)(t). Putting this together, system 100 can use thefollowing model:

P _(r)(kw _(d)(t)|q)=P _(r)(t|q)·P _(r)(d|q)  (1)

with slight abuse of the probabilistic notating this reflects theunderstanding that a token is a keyword in a document based on a queryif both the document is relevant to the query, and the token is relevantto the query.

To compute P_(r)(t|q), system 100 can estimate the probability that atoken is relevant to a query using a unigram language model.Essentially, system 100 can judge each token in the document accordingto the most similar token that appears in the search query. For a tokenin the document that has a match (a similar enough word in the querystring), the probability of relevance may be computed based on thissimilarity:

First, if token t appears in q, it can be said to be relevant to thequery, and system 100 can set P_(r)(t|q)=1. Otherwise, system 100 maycompute the relevance of t using distributed representation using anoise-contrastive estimation (NCE) word embedding model that was trainedon a corpus. NCE word embedding approximates the conditional probabilityof a term t, given a context term t_(i) byP(t|t_(i))=e^({dot over (t)}·{dot over (t)}i)·k², where k is the numberof negative samples and {dot over (t)} and {dot over (t)}_(i) are thevector representations of the target and context words, respectively.System 100 may use the combMAX aggregation method, and setP_(r)(t|q)=max_(tiEq)P(t|t_(i)). To reduce noise, system 100 may furtherapply a threshold of 0.5.

To compute P_(r)(d|q), system 100 can take into account the fact thatthe same query may be searched multiple times (possibly by differentusers), and may lead to multiple clicks on various documents. Denote byc(q,d) the count of clicks on document d as a result of querying q, andlikewise c(q)=Σ_(d∈D)c(q,d) and c(d)=Σ_(q∈Q)c(q,d), the total number ofclicks resulting from a query, and on a document respectively.Furthermore, define the “universe” of query q as the set of documentsclicked at least once following the query, that is: U_(q)={d|c(q,d)>0}.

The aim now, given the universe of a query, may be to determine when dappears relevant to q due to information content rather than erroneousor sporadic clicks. Intuitively, the larger c(q,d), the likelier it isthat q is indeed relevant to d. However, the number of clicks isaffected by the overall popularity of both the query and the document.Hence, system 100 can compute the p-value of the observed c(q,d) againsta null hypothesis where following a query, a set of documents ispresented and one is clicked randomly with probability proportional toits overall popularity. Thus, given a query q, the probability ofclicking on a document d under the null hypothesis isp_(d):=c(d)/(Σ_(d′∈Uq)c(d′)). This reduces to binomial hypothesistesting, which can be approximated by a normal distribution with meanc(q)·p_(d) and variance c(q)·p_(d)(1−p_(d)).

Finally, a token t may be detected as a keyword in a document due todifferent queries, with a goal being determining the probability of tbeing a keyword under any query. By assuming independence, system 100can compute the aggregated probability as:

P _(kw)(t)=1−Π_(q∈EQd)(1−P(k _(wd)(t)|q))  (2)

System 100 may handle missing data based on the assumption that the apriori probability of a token to be a keyword is independent from thenumber of clicks (popularity) of the document. However, there may be apositive correlation between document popularity and proportion ofkeywords. This correlation can be explained by the fact that unpopulardocuments lack queries that could provide evidence for tokens beingkeywords.

Accordingly, system 100 may introduce the notion of a yet unobservedquery which would lead to a click on a specific document and mark aspecific token as a keyword. System 100 may model the probability of theexistence of such a query by a decreasing sigmoid of the popularity ofthe document c(d), scaled to the range [0,μ] and having a central slopeof β, where μ is the a priori probability of being a keyword in thedataset. That is: f_(β)(c(d)):=2μ·(1−σ(β·c(d))) where σ(⋅) is thesigmoid function. This formulation lets the probability of being akeyword slide from p for a new document, to the observed value for awell-established (popular) document. This entails that the set ofrelevant queries is expanded by one, and Eq. 2 is modified accordingly:P_(kw)(t)=1−(1−f_(β)(c(d)))·Π_(q∈Qd)(1−P(k_(wd)(t)|q)). In order toestimate the values of p and β, system 100 may use the most populardocuments, assuming they have approximately full coverage of queries,leading to few missing keywords. Using this set, system 100 can computep. Under the described assumption, an optimal 13 will “correct” theprobabilities of the tokens, such that they will be independent to thepopularity of the documents. The estimated proportion of keywords iscomputed as:

μ_(d)(β)=(1/|d|)Σ_(t∈d)1−(1−f _(β)(c(d)))·Π_(q∈Qd)(1−P(k _(wd)(t)|q))

where |d| stands for the amount of tokens in document d. Using gradientdescent, system 100 may find a value β that minimizes the sum of squaredeviation of the estimated proportion of keywords from the true value μ:β*=arg max_(β)Σ_(d∈D)(μ_(d)(β)−μ)².

As post processing, system 100 may label consecutive stop words betweenkeywords as keywords with probabilities equal to the minimal probabilityof the non-stop-word tokens comprising it. Next, system 100 may omitremaining singleton keyphrases, unless they are derived from theidentical single-word query.

FIG. 6 shows document labeling processing 140 according to an embodimentof the present disclosure. System 100 can label document 102 using thekeyphrase determined as described above, which may allow document 102(with its label) to be used as training data for a supervised ML processthat produces a model for automatically identifying keyphrases forunlabeled documents, for example.

At 502, system 100 may compare the probability for the keyphrasedetermined as described above with a threshold probability value. If thekeyphrase has a probability of describing document 102 that is higherthan the threshold probability value, system 100 may determine that thekeyphrase is an accurate label for document 102.

At 504, system 100 may label document 102 with the keyphrase having aprobability greater than the threshold value. At 506, system 100 maystore the document label along with document 102 so that the labeleddocument may be used (e.g., for ML training or any other purpose).

FIG. 7 shows a computing device 700 according to an embodiment of thepresent disclosure. For example, computing device 700 may function asautomatic keyphrase labeling system 100 and/or as query system 504,machine learning system 200, any combinations thereof, or any portionsthereof. Computing device 700 may be implemented on any electronicdevice that runs software applications derived from compiledinstructions, including without limitation personal computers, servers,smart phones, media players, electronic tablets, game consoles, emaildevices, etc. In some implementations, computing device 700 may includeone or more processors 702, one or more input devices 704, one or moredisplay devices 706, one or more network interfaces 708, and one or morecomputer-readable mediums 710. Each of these components may be coupledby bus 712, and in some embodiments, these components may be distributedamong multiple physical locations and coupled by a network.

Display device 706 may be any known display technology, including butnot limited to display devices using Liquid Crystal Display (LCD) orLight Emitting Diode (LED) technology. Processor(s) 702 may use anyknown processor technology, including but not limited to graphicsprocessors and multi-core processors. Input device 704 may be any knowninput device technology, including but not limited to a keyboard(including a virtual keyboard), mouse, track ball, and touch-sensitivepad or display. Bus 712 may be any known internal or external bustechnology, including but not limited to ISA, EISA, PCI, PCI Express,NuBus, USB, Serial ATA or FireWire. In some embodiments, some or alldevices shown as coupled by bus 712 may not be coupled to one another bya physical bus, but by a network connection, for example.Computer-readable medium 710 may be any medium that participates inproviding instructions to processor(s) 702 for execution, includingwithout limitation, non-volatile storage media (e.g., optical disks,magnetic disks, flash drives, etc.), or volatile media (e.g., SDRAM,ROM, etc.).

Computer-readable medium 710 may include various instructions 714 forimplementing an operating system (e.g., Mac OS®, Windows®, Linux). Theoperating system may be multi-user, multiprocessing, multitasking,multithreading, real-time, and the like. The operating system mayperform basic tasks, including but not limited to: recognizing inputfrom input device 704; sending output to display device 706; keepingtrack of files and directories on computer-readable medium 710;controlling peripheral devices (e.g., disk drives, printers, etc.) whichcan be controlled directly or through an I/O controller; and managingtraffic on bus 712. Network communications instructions 716 mayestablish and maintain network connections (e.g., software forimplementing communication protocols, such as TCP/IP, HTTP, Ethernet,telephony, etc.).

Automatic keyphrase labeling instructions 718 may include instructionsthat enable computing device 700 to perform automatic keyphrase labelingsystem 100 functionality as described herein. Application(s) 720 may bean application that uses or implements the processes described hereinand/or other processes, for example applications used to provide thefunctionality of query system 14 and/or machine learning system 200. Theprocesses may also be implemented in operating system 714.

The described features may be implemented in one or more computerprograms that may be executable on a programmable system including atleast one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that can be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program may be written in anyform of programming language (e.g., Objective-C, Java), includingcompiled or interpreted languages, and it may be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructions mayinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors orcores, of any kind of computer. Generally, a processor may receiveinstructions and data from a read-only memory or a random access memoryor both. The essential elements of a computer may include a processorfor executing instructions and one or more memories for storinginstructions and data. Generally, a computer may also include, or beoperatively coupled to communicate with, one or more mass storagedevices for storing data files; such devices include magnetic disks,such as internal hard disks and removable disks; magneto-optical disks;and optical disks. Storage devices suitable for tangibly embodyingcomputer program instructions and data may include all forms ofnon-volatile memory, including by way of example semiconductor memorydevices, such as EPROM, EEPROM, and flash memory devices; magnetic diskssuch as internal hard disks and removable disks; magneto-optical disks;and CD-ROM and DVD-ROM disks. The processor and the memory may besupplemented by, or incorporated in, ASICs (application-specificintegrated circuits).

To provide for interaction with a user, the features may be implementedon a computer having a display device such as an LED or LCD monitor fordisplaying information to the user and a keyboard and a pointing devicesuch as a mouse or a trackball by which the user can provide input tothe computer.

The features may be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combinationthereof. The components of the system may be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a telephone network, aLAN, a WAN, and the computers and networks forming the Internet.

The computer system may include clients and servers. A client and servermay generally be remote from each other and may typically interactthrough a network. The relationship of client and server may arise byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other.

One or more features or steps of the disclosed embodiments may beimplemented using an API. An API may define one or more parameters thatare passed between a calling application and other software code (e.g.,an operating system, library routine, function) that provides a service,that provides data, or that performs an operation or a computation.

The API may be implemented as one or more calls in program code thatsend or receive one or more parameters through a parameter list or otherstructure based on a call convention defined in an API specificationdocument. A parameter may be a constant, a key, a data structure, anobject, an object class, a variable, a data type, a pointer, an array, alist, or another call. API calls and parameters may be implemented inany programming language. The programming language may define thevocabulary and calling convention that a programmer will employ toaccess functions supporting the API.

In some implementations, an API call may report to an application thecapabilities of a device running the application, such as inputcapability, output capability, processing capability, power capability,communications capability, etc.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example and notlimitation. It will be apparent to persons skilled in the relevantart(s) that various changes in form and detail can be made thereinwithout departing from the spirit and scope. In fact, after reading theabove description, it will be apparent to one skilled in the relevantart(s) how to implement alternative embodiments. For example, othersteps may be provided, or steps may be eliminated, from the describedflows, and other components may be added to, or removed from, thedescribed systems. Accordingly, other implementations are within thescope of the following claims.

In addition, it should be understood that any figures which highlightthe functionality and advantages are presented for example purposesonly. The disclosed methodology and system are each sufficientlyflexible and configurable such that they may be utilized in ways otherthan that shown.

Although the term “at least one” may often be used in the specification,claims and drawings, the terms “a”, “an”, “the”, “said”, etc. alsosignify “at least one” or “the at least one” in the specification,claims and drawings.

Finally, it is the applicant's intent that only claims that include theexpress language “means for” or “step for” be interpreted under 35U.S.C. 112(f). Claims that do not expressly include the phrase “meansfor” or “step for” are not to be interpreted under 35 U.S.C. 112(f).

What is claimed is:
 1. A method of automatic keyphrase labelingcomprising: extracting, by a processor, a plurality of keywords from atleast one search query that resulted in a selection of a documentincluded in a search result; for each of the plurality of keywords,determining, by the processor, a probability that the keyword describesthe document; generating, by the processor, one or more keyphrases, thegenerating comprising selecting each of the plurality of keywords havinga probability greater than a predetermined threshold value for insertioninto at least one of the one or more keyphrases and assembling the oneor more keyphrases from the selected plurality of keywords; labeling, bythe processor, the document with the one or more keyphrases; andstoring, by the processor, the labeled document as training data in adocument corpus.
 2. The method of claim 1, wherein the extractingcomprises removing at least one uninformative portion from the at leastone search query, wherein the plurality of keywords comprises a portionof the at least one search query that remains after the at least oneuninformative portion is removed.
 3. The method of claim 2, wherein theassembling comprises: assigning a probability to each word in the atleast one uninformative portion; and incorporating each word withprobability assigned into the keyphrase for the document.
 4. The methodof claim 1, wherein the determining comprises determining an affinitybetween the search query and the document, the affinity forming acomponent of each probability of each respective keyword.
 5. The methodof claim 4, wherein determining the affinity comprises: determining anexpected random distribution of selections of the document in the searchresult among selections of each of a plurality of documents appearing inthe search result, wherein the document is one of the plurality ofdocuments; determining an actual distribution of selections of thedocument; and setting the affinity based on a proportion of the actualdistribution to the expected random distribution.
 6. The method of claim1, wherein the determining comprises, for each of the plurality ofkeywords, determining a keyword component of the probability based on acomparison between the keyword and content of the document.
 7. Themethod of claim 6, wherein the determining comprises calculating theprobability using: an affinity between the search query and thedocument; and the keyword component of the probability.
 8. The method ofclaim 1, wherein: the assembling comprises determining an overallprobability that the keyphrase describes the document; and the labelingcomprises determining that the overall probability is greater than apredetermined overall threshold value.
 9. The method of claim 1, furthercomprising training, by the processor, a supervised machine learningmodel using the training data.
 10. A system configured to performautomatic keyphrase labeling comprising: at least one memory configuredto store at least one search query; and a processor in communicationwith the at least one memory and configured to: extract a plurality ofkeywords from the at least one search query that resulted in a selectionof a document included in a search result; for each of the plurality ofkeywords, determine a probability that the keyword describes thedocument; generate one or more keyphrases by performing processingincluding selecting each of the plurality of keywords having aprobability greater than a predetermined threshold value for insertioninto at least one of the one or more keyphrases and assembling the oneor more keyphrases from the selected plurality of keywords; label thedocument with the keyphrase; and store the labeled document as trainingdata in a document corpus in the at least one memory.
 11. The system ofclaim 10, wherein the processor is configured to extract the pluralityof keywords by performing processing comprising removing at least oneuninformative portion from the at least one search query, wherein theplurality of keywords comprises a portion of the at least one searchquery that remains after the at least one uninformative portion isremoved.
 12. The system of claim 11, wherein the processor is configuredto assemble the keyphrase by performing processing comprising: assigninga probability to each word in the at least one uninformative portion;and incorporating each word with probability assigned into the keyphrasefor the document.
 13. The system of claim 10, wherein the processor isconfigured to determine each probability by performing processingcomprising determining an affinity between the search query and thedocument, the affinity forming a component of each probability of eachrespective keyword.
 14. The system of claim 13, wherein determining theaffinity comprises: determining an expected random distribution ofselections of the document in the search result among selections of eachof a plurality of documents appearing in the search result, wherein thedocument is one of the plurality of documents; determining an actualdistribution of selections of the document; and setting the affinitybased on a proportion of the actual distribution to the expected randomdistribution.
 15. The system of claim 10, wherein the processor isconfigured to determine each probability by performing processingcomprising, for each of the plurality of keywords, determining a keywordcomponent of the probability based on a comparison between the keywordand content of the document.
 16. The system of claim 15, wherein theprocessor is configured to determine each probability by performingprocessing comprising calculating the probability using: an affinitybetween the search query and the document; and the keyword component ofthe probability.
 17. The system of claim 10, wherein: the processor isconfigured to assemble the keyphrase by performing processing comprisingan overall probability that the keyphrase describes the document; andthe processor is configured to label the document by performingprocessing comprising determining that the overall probability isgreater than a predetermined overall threshold value.
 18. The method ofclaim 17, wherein the processor is configured to train a supervisedmachine learning model using the labeled document as training data. 19.A method of automatic keyphrase labeling and machine learning trainingcomprising: extracting, by a processor, a plurality of keywords from atleast one search query that resulted in a selection of a document of aplurality of documents included in a search result, the extractingcomprising removing at least one uninformative portion from the at leastone search query, wherein the plurality of keywords comprises a portionof the at least one search query that remains after the at least oneuninformative portion is removed; determining, by the processor, anaffinity between the search query and the document, wherein determiningthe affinity comprises: determining an expected random distribution ofselections of the document among selections of any of the plurality ofdocuments; determining an actual distribution of selections of thedocument; and calculating the affinity based on a proportion of theactual distribution to the expected random distribution; for each of theplurality of keywords, determining, by the processor, a probability thatthe keyword describes the document, the determining comprising:determining a keyword component of the probability based on a comparisonbetween the keyword and content of the document; and calculating theprobability using: the affinity between the search query and thedocument; and the keyword component of the probability; generating, bythe processor, one or more keyphrases, the generating comprising:selecting each of the plurality of keywords having a probability greaterthan a predetermined threshold value for insertion into at least one ofthe one or more keyphrases; assigning a probability to each word in theat least one uninformative portion; and assembling the one or morekeyphrases from the selected plurality of keywords and each word in theat least one uninformative portion with probability assigned into thekeyphrase for the document; determining, by the processor, an overallprobability that the keyphrase describes the document; labeling, by theprocessor, the document with the keyphrase, the labeling comprisingdetermining that the overall probability is greater than a predeterminedoverall threshold value; and training, by the processor, a supervisedmachine learning model using the labeled document as training data. 20.The method of claim 19, further comprising determining, by theprocessor, a label for an unlabeled document by performing machinelearning processing using the trained supervised machine learning model.